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I.  INTRODUCTION 


A.  EVALUATION  OBJECTIVES 

In  February  1983,  an  automated  Tri-Service  Patient  Appointment 
and  Scheduling  (TRIPAS)  System  (initial  capability)  replaced  the 
manual  appointment  system  at  the  USAF  Medical  Center,  Keesler  Air 
Force  Base  (Keesler),  Biloxi,  Mississippi.  This  Installation  was 
carried  out  as  part  of  a  contract  from  the  Tri-Service  Medical 
Information  Systems  (TRIMIS)  Program  Office  (TPO).  The  experience 
with  the  implementation  of  this  system  is  being  evaluated  in  order  to 
provide  TPO  with  information  concerning  the  costs  and  benefits  of 
providing  automated  support  to  patient  appointment  and  scheduling  in 
Military  Treatment  Facilities  (MTFs) . 

The  purpose  of  this  overall  evaluation  effort,  both  baseline 
(manual  system)  and  postimplementation  (automated  system) ,  is  to 
assess: 

•  Performance  and  impact  of  the  TRIPAS  System;  and 

•  Desirable  improvements  in  the  TRIPAS  System. 

The  impact  of  the  TRIPAS  System  is  measured  by  documenting 
changes  in  the  performance  of  the  appointment  and  scheduling  system 
between  the  baseline  and  postimplementation  periods  (see  refs.  3,  6). 
The  comparison  is  accomplished  both  by  measurement  of  performance 
indicators  and  by  qualitative  assessment  of  patient  and  staff 
perceptions  and  acceptance  of  the  system.  Results  are  used  to 
determine  to  what  extent  the  TRIPAS  System  has  met  or  exceeded  user 
expectations  and  goals,  and  to  provide  data  useful  in  revising  the 
economic  analysis  of  the  system.  Also,  the  evaluation  results  will  be 
used  to  identify  system  and  operational  changes  that  would  lead  to 
realizing  additional  benefits  from  system  use. 
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The  overall  objective  of  the  TRIMIS  Program  Office  in  funding 
this  evaluation  is  to  provide  accurate  information  concerning  the 
TRIPAS  System's  costs  and  benefits.  This  information  is  used  to  make 
decisions  regarding  future  appointment  and  scheduling  applications. 


The  evaluation  of  the  manual  and  automated  patient  appointment 
and  scheduling  systems  at  Keesler  is  being  documented  in  the  following 
sequence: 


•  A  preliminary  economic  analysis  was  performed  by  the 
TRIMIS  Program  Office  (1). 

•  A  plan  for  conducting  baseline  data  collection  and 
analysis  was  prepared  (2). 

•  Analysis  of  the  baseline  (manual  system)  results  was 
presented  in  a  report  (3). 

•  Following  installation,  the  implementation  of  the  TRIPAS 
System  was  monitored  until  it  was  deemed  ready  for  a 
postimplementation  evaluation.  An  Interim  Status  Report 
on  implementation  status  was  submitted  to  TPO  (4). 

•  A  plan  for  conducting  data  collection  and  analysis  in  the 
postimplementation  period  was  drafted  (5). 

•  Results  of  data  collection  for  the  automated  period, 
along  with  a  comparison  of  results  from  the  two  periods, 
were  presented  (6). 

•  The  final,  updated  Economic  Analysis  was  presented  in 
draft  form  on  September  15,  1983  (7). 

This  report  presents  the  plan  for  data  collection  in  the 
postimplementation  period,  revised  in  accordance  with  the  results  of 
the  data  analysis. 


B.  SETTING 


USAF  Medical  Center  Keesler  is  designated  as  an  area  medical  and 
referral  center,  providing  health  care  to  beneficiaries  residing 
throughout  the  southeastern  region  of  the  United  States.  To  accom¬ 
plish  this  mission,  the  medical  center  currently  operates  a  325-bed 
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medical  treatment  facility,  a  50-bed  aeromedical  staging  flight,  a 
78-room  dental  treatment  facility,  and  40  outpatient  clinics.  The 
medical  center  includes  approximately  325  officers,  800  enlisted 
personnel,  and  230  civilians.  In  addition,  an  average  of  105  officers 
and  30  enlisted  personnel  are  assigned  to  the  Center  for  specialized 
training. 

Representative  average  monthly  volumes  at  this  facility  include 
950  inpatient  admissions,  34,300  outpatient  visits,  and  2,250  dental 
visits.  I 

C.  TRIPAS  SYSTEM  AT  KEESLER  AFB 


The  Tri-Service  Patient  Appointment  and  Scheduling  System  | 

> 

provides  state-of-the-art  automated  support  to  clinic  appointment  J 

scheduling.  The  functions  supported  include  booking,  cancelling  and 
verifying  appointments;  monitoring  provider  schedules;  maintaining  the 
registration  data;  generating  medical  record  pull  lists;  and  I 

i 

generating  reports. 

TRIPAS  is  an  adaptation  of  an  automated  patient  appointment 
system,  the  software  for  which  was  developed  by  Medical  Data 
Corporation  and  acquired  by  HB0  of  Atlanta,  Georgia.  Martin  Marietta, 
of  Bethesda,  Maryland,  is  the  prime  contractor,  with  HBO  the 
subcontractor  responsible  for  developing  and  supporting  system 
software.  TRIPAS  software  is  written  in  standard  MUMPS. 

There  are  three  sizes  of  CPU  hardware  configuration:  large, 
medium,  and  small.  Keesler  received  the  large  configuration,  which 
includes  1  Central  Processing  Unit  (CPU),  1  Line  Printer,  20  Character 
Printers,  66  Cathode  Ray  Tube  (CRT)  terminals,  and  2  Automated 
Embossing  Machines. 
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Seven  CRTs  are  located  in  the  Central  Appointment  Section  (CAS) , 
and  most  outpatient  clinics  have  at  least  one  terminal.  The  CPU  is 
housed  in  a  specially  secured  and  air  conditioned  computer  room  that 
also  houses  the  CPUs  for  the  TRIPAD  and  TRIPHARM  Systems  and  will 
house  the  TRILAB  and  CAPOC  Systems'  CPUs. 

D.  ORGANIZATION  OF  THE  REPORT 


This  report  consists  of  two  major  sections,  including  this 
Introduction.  The  second  section  discusses  the  methodology  proposed 
for  the  postimplementation  evaluation  of  the  TRIPAS  system  at  Keesler. 
It  is  divided  into  four  sections:  approach,  data  collection,  data 
analysis,  and  schedule. 

References  follow  the  methodology  section.  Data  collection 
instruments  proposed  to  be  used  for  this  evaluation  are  included  as  an 
appendix  to  the  report. 
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II.  METHODOLOGY 


A.  APPROACH 


The  overall  evaluation  of  the  TRIPAS  System  is  a  "before  and 
after"  study  of  the  impact  of  TRIPAS  on  the  effectiveness  and  cost  of 
outpatient  appointment  and  scheduling  services  at  Keesler.  In 
addition  to  the  assessment  at  Keesler,  focused  evaluations  have  been 
conducted  at  Brooke  Army  Medical  Center  and  Ehrling  Bergquist  USAF 
Regional  Hospital  to  assess  preliminary  indicators  of  the  functional 
performance  and  impact  of  the  TRIPAS  System  and  to  identify  additional 
potential  impacts  (8).  The  evaluation  at  Keesler  adheres  to  the 
methodology  used  at  these  two  previous  sites.  The  system's 
implementation  was  monitored  during  a  site  visit  (April  11-12,  1983); 
interviews  were  conducted  with  Keesler  'staff,  before  and  2  months 
after  the  system  was  implemented.  These  assessments  led  to  the 
identification  of  the  following  anticipated  impacts  of  TRIPAS  on 
appointment  and  scheduling  operations  at  Keesler: 

•  Increased  availability  of  appointments; 

•  Reduction  in  the  time  required  to  make,  change,  verify, 
or  cancel  an  appointment; 

•  Reduced  waiting  times  within  clinics,  and  in  the  time 
period  from  initial  entry  into  the  medical  care  system  to 
diagnosis  and  treatment  in  the  appropriate  clinic; 

•  Increased  availability  of  medical  records  and  x-rays; 

•  Reduction  in  the  number  of  patients  who  miss  their 

appointments ; 

•  Increased  patient  and  staff  satisfaction  with  the 
appointment  and  scheduling  system; 

•  Optimization  of  care  provider  productivity; 

•  Reduction  in  the  effort  required  to  make  or  change 

provider  schedules; 
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•  Reduction  in  appointment  transcription  and  processing 
errors; 

•  Reduced  cost  per  appointment  transaction  and  outpatient 
visit; 

•  Increased  ability  to  accommodate  same-day  multiple-clinic 
appointments ;  and 

•  Enhanced  generation  of  useful  management  reports. 

These  areas  of  projected  impact  are  intended  to  be  responsive  to  the 
goals  and  objectives  of  the  TRIPAS  System  as  set  forth  by  the  TR1MIS 
Medical  Review  Group  (Exhibit  1).  From  these  likely  impacts,  a  set  of 
performance  criteria  have  been  established  (Exhibit  2).  At  the  time 
of  the  implementation  monitoring  visit  in  April  1983,  It  was 
determined  that  13  of  these  19  criteria  had  been  met,  and  the  system 
was  considered  to  be  ready  for  postimplementation  evaluation. 

The  evaluation  focuses  on  these  impacts.  In  addition  to 
the  operation  of  the  Central  Appointment  Section  (CAS),  five 
outpatient  clinics  were  selected  for  study:  two  that  use  CAS  to  make 
appointments  and  three  that  book  their  own  appointments.  In  one  of 
the  latter,  the  Dental  Clinic,  only  provider  and  patient  levels  of 
satisfaction  are  assessed.  The  appointment  process,  clinic  workload, 
and  number  of  assigned  providers  are  detailed  as  follows  (10): 

Total  Annual  Number  of  As- 

Process  and  Clinic  Outpatient  Visits  signed  Providers 

Uses  CAS: 


Primary  Care  (PC) 

51,000 

15 

Internal  Medicine  (IM) 

8,000 

45 

Books  Own  Appointments: 

General  Surgery  'GS) 

29,100 

25 

Orthopedics  (0) 

13,200 

5 

Dental  (D) 

26,400 

22 
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EXHIBIT  I 


TRIPAS  SYSTEM  GOALS  AND  OBJECTIVES 


1.  Reduced  time  to  make  a  patient  appointment. 

2.  Reduced  clinic  waiting  time. 

3.  Reduced  trips  to  receive  care  in  multiple 
clinics. 

4.  Reduced  appointment  clerical  work. 

5.  Increased  utilization  by  better  matching 
patients,  providers,  facilities. 

6.  Reduced  "no-shows." 

7.  Reduced  appointment  transcription  and  process¬ 
ing  errors. 

8.  Improved  interfacility  communication. 

9.  Enhanced  decision  making. 

10.  Establishment  of  a  data  bank. 


SOURCE:  TRIMIS  Medical  Review  Group,  Initial  Project  Objectives 
and  Evaluation  Criteria,  TRIMIS  Program  Office,  Bethesda, 
Maryland,  April  1978 
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EXHIBIT  2 


TRIPAS  SYSTEM  PERFORMANCE  CRITERIA 


A.  System  Operation/Structure 

1.  TRIPAS  Uptime 

2.  Clerks  and  Technicians  Trained 

3.  Terminals  in  Use 

4.  Printers  in  Use 

5.  Patient  Population  Registered 

6.  Provider  Schedules  Loaded 

7.  Pre-existing  Appointments  Loaded 

B.  System  Use/Process 

8.  Clerks  and  Technicians  Experienced  in  TRIP A: 

Use 

9.  Users  Meeting  Operational  Standards  for  Quality 

10.  Users  Meeting  Operational  Standards  for  Time 

11.  Use  of  Software  Modules,  Options,  and  Functions 

12.  Manual  Tasks  Supported  by  Software  Modules, 
Options,  and  Functions 

C.  System  Benefits/Outcome 

13.  Clerk  and  Technician  Productivity  I 
(seconds/transaction) 

14.  Clerk  and  Technician  Productivity  II 
(transactions/hour) 

15.  Average  Waiting  Time  for  Telephone  To  Be 
Answered  (Peak  Time) 

16.  Clinic  No-Shows 

17.  Satisfaction  with  TRIPAS  System  Performance 
(Staff) 

18.  Satisfaction  with  Appointment  and  Scheduling 
System  (Patients) 

19.  Satisfaction  with  TRIPAS  Reports  and  Products 
(Supervisors  and  NCOIC’s) 
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The  study  clinics  have  been  selected  on  the  basis  of  their 
scheduling  methods  and  the  volume  of  patients,  especially  active-duty 
personnel,  seen.  Though  the  evaluation  at  Keesler  involves  a  sample 
of  all  patients  who  use  the  clinics  under  study,  particular  attention 
is  given  to  the  impact  of  TRIPAS  on  appointment  and  scheduling 
activities  for  active-duty  personnel,  since  time  savings  among 
active-duty  personnel  results  in  direct  benefit  to  the  DoD. 

The  proposed  study  design  will  reveal  the  effect  of  TRIPAS  on 
patient  appointment  and  scheduling  activities,  particularly  on  CAS  and 
clinic  productivity,  scheduling  efficiency,  and  impact  of  improved 
scheduling  on  provider  productivity.  The  study  will  also  track  the 
continuity  of  patient  care  from  primary  entry  through  specialized 
clinic  responses. 

The  impact  areas  that  will  be  examined,  the  measurement 
techniques,  the  sites  of  data  collection,  and  the  observation  periods 
are  displayed  in  Exhibit  3.  Sample  sizes  are  designed  to  detect 
changes  in  performance  that  are  statistically  significant  at  0.05. 
The  amount  of  data  collected  for  each  impact  area  and  site  within 
Keesler  matches  the  amount  of  data  (sample  sizes)  collected  during  the 
baseline  period. 

In  order  to  optimize  the  use  of  data  collection  time  and 
personnel,  data  will  be  collected  at  each  of  the  four  sample  clinics 
and  CAS  at  the  same  time,  for  10  days  each.  Clinic  and  CAS  data  will 
be  collected  by  sampling  approximately  10  calls  per  hour  at  each 
location. 

B.  DATA  COLLECTION 

NCSI  personnel  and  on-site  data  collectors  will  monitor 
appointment  activities  for  10  working  days  in  the  Central  Appointments 
desk  and  in  four  clinics:  Primary  Care,  Internal  Medicine, 
Orthopedics,  and  General  Surgery.  (The  Dental  Clinic  will  be  surveyed 
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over  two  working  days,  with  questionnaires  administered  to  providers 
and  patients.)  All  data  will  be  recorded  by  evaluation  team  members 
or  hired  data  collectors,  with  the  exception  of  cost  data,  which  will 
be  obtained  from  the  TRIPAS  contract,  and  reports  concerning 
availability  of  clinic  appointments,  which  will  be  obtained  from 
existing  hospital  reports.  Prior  to  postimplementation  data 
collection,  on-site  data  collectors  will  be  briefed  by  NCSI  staff  on 
data  collection  procedures,  given  a  demonstration,  and  then  checked 
for  proficiency  and  accuracy  as  they  perform  the  data  collection 
tasks.  The  same  four  data  collectors  used  in  the  baseline  evaluation 
will  collect  data  in  the  postimplementation  period.  Each  of  these 
collectors  is  a  Red  Cross  volunteer  at  Keesler  and  is  thoroughly 
familiar  with  hospital  operations. 

The  rest  of  this  section  on  data  collection  will  consider  the 
potential  impacts  of  the  TRIPAS  System  individually  and  will  examine 
measurement  techniques  to  be  employed.  Copies  of  all  data  collection 
instruments  referred  to  in  this  section  are  found  in  the  Appendix. 

1.  Total  Time  per  Appointment  Transaction 

In  order  to  measure  the  amount  of  time  spent  in  the  appointment 
and  scheduling  process,  timed  observations  of  appointment  activities 
will  be  performed.  The  time  required  to  complete  appointment-related 
phone  transactions  will  be  measured,  as  will  the  amount  of  time 
required  to  reach  a  CAS  clerk  on  the  telephone.  Data  collectors  will 
randomly  time  telephone  calls  to  CAS  and  two  clinics  that  book  their 
own  appointments  (General  Surgery  and  Orthopedics)  using  the  Telephone 
Interaction  Time  survey  instrument.  Ten  calls  per  hour  will  be  timed 
in  CAS,  and  at  least  5  per  hour  in  the  clinics,  for  a  period  of  10 
days.  Measurement  of  the  queue  to  reach  a  clerk  in  CAS  will  be 
accomplished  by  use  of  the  automated  telephone  monitoring  system, 
which  permits  the  supervisor  of  CAS  to  assess  the  number  of  callers  in 
the  queue  and  the  amount  of  time  the  next  caller  to  be  answered  has 
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been  waiting.  Data  collectors  will  query  the  telephone  monitoring 
equipment  once  every  half-hour  from  0730  to  1600  for  a  10-day  period 
and  will  record  the  information  on  the  CAS  Queue  Record. 


The  Outpatient  Waiting  Time  Record  will  be  distributed  in  the 
four  sample  clinics:  Internal  Medicine,  Primary  Care,  General  Surgery, 
and  Orthopedics.  The  questionnaire  is  self-administering  and  records 
the  time  that  a  patient  arrives,  the  scheduled  time  of  the 
appointment,  and  the  tine  that  the  patient  is  actually  seen  by  a 
health  care  provider.  The  questionnaire  will  be  distributed  in  each 
clinic  at  a  rate  of  12-15  per  day  for  10  days  in  order  to  collect  a 
total  of  125. 


3.  Clinic  Referrals 

Data  collectors  will  administer  the  Patient  Tracking  Form  in  the 
General  Surgery  and  Orthopedic  clinics  in  order  to  assess  the 
specialty  clinic  referral  process.  The  Patient  Tracking  Form  consists 
of  a  brief  interview  covering  items  such  as  the  time  required  to  be 
booked  and  seen  in  the  clinic  after  contact  with  Primary  Care  as  well 
as  patient  satisfaction  with  the  process.  Approximately  30 
questionnaires  will  be  collected  In  each  of  the  two  clinics,  divided 
equally  between  active  and  non-active  duty  patients.  The  fact  that 
patients  will  be  tracked  continually,  from  their  initial  entry  into 
the  medical  care  system  to  their  diagnosis  and  treatment  in  the 
appropriate  referral  clinic,  will  facilitate  the  assessment  of  the 
availability  of  appointments  and  physicians. 


U.  CAS  Workload 

Two  measures  will  help  determine  the  CAS  workload.  A  log 
(Telephone  Log)  kept  by  each  CAS  clerk  will  record  all  informa¬ 
tion  calls  and  appointment-unavailable  calls.  These  data  will 


/ti  Arthur  D.  Little,  Inc. 


12 


be  combined  with  TRIPAS  statistical  reports  that  show  the  number  of 
appointments .booked  and  appointments  cancelled  by  each  clerk  in  order 
to  calculate  the  total  transaction  workload.  Data  will  be  collected 
for  10  days. 

5 .  Provider  Productivity 

The  Provider  Productivity  Record  will  be  given  to  eight  doctors, 
two  for  each  sample  clinic,  in  order  to  assess  the  rate  of 
patient/provider  encounters.  The  form  will  be  completed  by 
non-resident  doctors  who  regularly  see  most  of  their  patients  in  the 
sample  clinic.  By  recording  the  number  of  patients  seen  and  the 
amount  of  time  available  to  each  doctor  to  see  clinic  patients,  the 
degree  of  service  utilization  will  be  measured. 

6.  Clinic  Workload 


In  order  to  determine  quantitative  information  for  clinic 
workloads,  each  of  the  four  sample  clinics  will  be  monitored  for  10 
days  concurrently  using  the  Daily  Clinic  Record.  This  form  will  be 
completed  by  clinic  personnel,  with  help  from  data  collectors,  and 
will  record  all  scheduled  appointments,  walk-ins,  cancellations,  and 
no-shows.  Availability  of  medical  records  and  x-rays  will  also  be 
measured.  The  aforementioned  Patient  Waiting  Time  Record  collects 
data  on  the  availability  of  medical  records  and  x-rays  in  addition  to 
waiting  time  data. 

7.  Number  of  Transcriotion  Errors 


The  number  of  transcription  errors  made  on  appointment  lists  will 
be  kept  on  the  Daily  Clinic  Record  for  10  days  in  the  four  sample 
clinics.  Transcription  errors  will  be  counted  when  a  patient  arrives 
for  an  appointment  who  is  not  recorded  in  the  proper  slot  on  the 
appointment  list. 
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8.  Patient  Perception  of  Scheduling  System 

Data  collectors  will  distribute  self-administering  questionnaires 
to  patients  in  order  to  gather  data  on  patient  opinions  of  the 
appointment  and  scheduling  system.  The  Outpatient  Questionnaire  asks 
for  objective  information  concerning  the  patient's  experience  with  the 
appointment  system  in  addition  to  a  section  on  patient  satisfaction 
with  the  various  aspects  of  the  system.  The  Outpatient  Questionnaire 
will  be  distributed  in  the  four  sample  clinics  at  the  rate  of  12-15 
per  day  for  10  days.  An  abbreviated  form  (Patient  Questionnaire, 
Dental  Clinic)  including  only  satisfaction  ratings  will  be  given  to 
100  patients  during  2  days  at  the  Keesler  Dental  Clinic. 


Staff  Perception  of  Scheduling  System 


Surveys  (questionnaires  and  interviews)  of  patients,  care 
providers,  appointment  scheduling  and  support  staff,  and 
administrators  will  be  used  to  measure  satisfaction  with  automated 
services.  The  surveys  will  contain  closed-end  questions  concerning 
satisfaction  with  various  aspects  of  the  appointment  and  scheduling 
system  and  closed-end  questions  concerning  performance  of  the  system. 


a.  Telephone  clerks  and  receptionists.  The  Appointment  and 
Scheduling  Support  Staff  Questionnaire  will  be  administered  to  all 
appointment  clerks  in  CAS  and  the  four  sample  clinics. 


b.  Providers.  The  Health  Care  Provider  Questionnaire  will  be 
administered  to  all  providers  in  the  four  sample  clinics  and  in  the 
dental  clinic.  A  Provider  Interview  Form  will  be  used  to  solicit 
opinions  from  two  providers  at  each  sample  clinic  in  regard  to  TRIPAS 
System  impact. 

c.  Supervisors  and  administrators.  Supervisory  and 
administrative  personnel  will  be  interviewed  in  several  locations: 
CAS,  four  sample  clinics,  Hospital  Administration  (ADM)  and  Patient 
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Affairs  (PA),  and  Medical  Records.  At  least  one  individual 
responsible  for  each  location  will  be  interviewed  about  the  TRIPAS 
System  reports  they  receive. 

10.  Cost  of  Scheduling  System 

In  the  postimplementation  period,  a  review  of  (1)  Keesler 
financial  records  of  personnel  and  their  salaries,  of  equipment  and 
supplies,  and  of  base  overhead  support,  and  (2)  TRIPAS  System 
procurement  costs  will  provide  cost  data  from  which  the  cost  per  unit 
of  workload  (cost  per  appointment,1*  cost  per  appointment  transaction) 
will  be  determined. 

C.  ANALYSIS 

Analysis  of  the  data  will  be  of  two  general  types: 

•  Comparisons  of  postimplementation  results  with  baseline 
results  for  the  same  impact;  and 

•  Comparisons  of  baseline  and  postimplementation  results 
with  performance  goals  set  by  Keesler  staff,  and  with 
system  goals  and  objectives  set  by  the  Medical  Review 
Group . 

For  both  types  of  analysis,  quantified  baseline  and  postimplementation 
results  will  be  presented  side  by  side  and  improvement  or 
deterioration  in  performance  discussed.  Results  will  be  analyzed  for 
each  clinic  and  CAS  as  well  as  aggregated  for  all  clinics.  In 
general,  impacts  will  be  clustered  in  one  of  five  sections  of  this 
report: 

•  Requesting  an  appointment; 

•  Being  scheduling  for  an  appointment; 

•  Having  the  medical  records  located  and  sent  to  the 
appropriate  clinic; 

15 

/ti  Arthur  D.  Little,  Inc. 


•  Receiving  care  at  Che  appointed  time;  or 

•  Cost -of  appointment  and  scheduling  operations. 

So  that  results  can  be  understood  and  interpreted  in  the  context 
of  Keesler  operations,  each  of  these  sections  will  contain  a 
description  of  the  process;  quantified  time-  and  workload-related 
measurements;  care  provider,  appointment  staff,  and  patient  opinions; 
and  lastly,  conclusions  drawn  from  the  results  obtained. 


Where  the  differences  between  observation  periods  require 
interpretation,  statistical  tests  (chi-square,  t-tests)  will  be 
performed  to  assess  the  significance  of  differences  between 
performance  periods.  Differences  in  average  clinic  waiting  time  to 
see  a  care  provider,  for  example,  will  be  considered  significant  if 
the  probability  of  the  difference  occurring  by  change  is  less  than  one 
in  twenty  (p  <  0.05).  The  data  compared  for  quantified  impacts,  in 
general,  will  involve  stratified  means  and  standard  deviations. 

Questionnaire  data  will  be  keypunched,  edited,  and  analyzed  using 
a  standard  statistical  computer  program.  A  Likert  scale  will  be  used 
to  convert  responses  concerning  importance  of  and  satisfaction  with 
the  appointment  system  into  mean  ratings. 

The  performance  goals  defined  by  Keesler  staff  are  set  for  three 
levels: 


•  Minimum  acceptable. 

•  Average  expected. 

•  Maximum  attainable. 

Comparison  of  baseline  and  postimplementation  results  with  these 

levels  of  anticipated  performance  (as  determined  in  a  provider  survey) 
will  allow  an  assessment  of  how  well  the  TRIPAS  System  meets  user 
expectations.  For  example,  in  the  baseline  period  62%  of  care 

providers  expressed  satisfaction  with  the  overall  quality  of 
appointment  and  scheduling  services.  They  also  felt  that  56%  was  the 
minimum  acceptable  goal  for  -care  provider  satisfaction  and  that  75% 
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was  che  average  expected  value.  If  this  figure  is  greater  than  75%  in 

the  postimp  lamentation  period,  then  it  can  be  concluded  that  the  > 

TRIPAS  System  has  Improved  satisfaction  of  care  providers  past  the 

average  level  they  expect  for  appointment  and  scheduling  operations. 

These  analyses  provide  a  second  method  of  interpreting  the 
significance  of  changes  in  performance  between  the  manual  and 
automated  system.  (Performance  goals  have  been  established  for  about 
one-third  of  the  TRIPAS  impacts  that  will  be  assessed.) 
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Evaluation  Plan  (Period  X)  for  the  Patient  Appointment  and 
Scheduling  System  at  USAF  Medical  Center,  Keesler  Air  Force  Base. 
Draft  report.  9/30/82. 

3.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 

Baseline  Evaluation  of  the  Tri-Service  Patient  Appointment  and 
Scheduling  System  at  USAF  Medical  Center,  Keesler  Air  Force  Base. 
Final  report  for  period  9/17/82-8/31/83.  8/31/83. 

4.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 

Implementation  Monitoring  of  the  Patient  Appointment  and 
Scheduling  System  at  USAF  Medical  Center,  Keesler  Air  Force  Base. 
4/29/83. 


5.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 
Evaluation  Plan  (Period  Y)  for  the  Patient  Appointment  and 
Scheduling  System  at  USAF  Medical  Center,  Keesler  Air  Force  Base. 
Draft  plan.  5/12/83. 


6.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 

]  Evaluation  of  the  Tri-Service  Patient  Appointment  and  Scheduling 

J  System  at  USAF  Medical  Center,  Keesler  Air  Force  Base.  Draft 

final  report  for  period  2/16/82-8/26/83.  8/26/83. 

7.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 

Economic  Analysis  of  the  Tri-Service  Appointment  and  Scheduling 

System  ( TRIP AS ) .  Draft  economic  analysis  for  period 

2/17/82-9/15/83.  9/15/83. 


8.  National  Capitol  Systems,  Inc.,  and  Arthur  D.  Little,  Inc. 
Report  of  Interim  Evaluation  of  the  Patient  Appointment  and 
Scheduling  System  at  Brooke  Army  Medical  Center  and  Ehrling 
Bergquist  USAF  Regional  Hospital.  Interim  Status  Report.  10/82. 

9.  TRIMIS  Medical  Review  Group.  Initial  Project  Objectives  and 
Evaluation  Criteria.  TRIMIS  Program  Office,  Bethesda,  Maryland, 
April  1978. 

10.  Survey  of  clinic  workload  statistics  conducted  by  Keesler 
hospital  services. 
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APPENDIX : 


DATA  COLLECTION  INSTRUMENTS 


Outpatient  Questionnaire 

Patient  Questionnaire,  Dental  Clinic 

CAS  Queue  Record 

Telephone  Interaction  Queue  Time 
Daily  Clinic  Record 
Patient  Waiting  Time  Record 
Telephone  Log 

Health  Care  Provider  Questionnaire 

Appointment  and  Scheduling  Support 
Staff  Questionnaire 

Phone  Consult  Record 

Provider  Productivity  Record 

Provider  Interview  Form 

Patient  Tracking  Form 


A-l 
A-4 
A- 5 
A- 6 
A- 7 
A- 8 
A- 9 
A-10 

A-14 
A-l  7 
A-18 
A-19 
A-21 


/k  Arthur  D.  Little,  Inc. 


OUTPATIENT  QUESTIONNAIRE 

Cl inic _ 

Date__ _ 

Tine _ 

We  ask  you  Co  Cake  a  few  ninuCes  co  answer  some  ques cions  abouC  our 
pacienc  care  and  che  appoincmenc  scheduling  syscen  here.  Infornacion 
from  chis  quescionnaire  will  help  us  provide  che  besc  possible  nedical 
care.  All  responses  will  be  held  in  scriccest  confidence. 

1.  Is  chis  your  firsc  visit  co  this  clinic?  (check  one) 

1.  _ yes 

2.  no 

3 .  can1 t  remember 

2.  How  would  you  describe  your  need  for  chis  appoincmenc?  (check  one) 

1.  _ emergency  (needed  co  be  seen  on  sane  day) 

2.  _ semi-urgenc  (needed  co  be  seen  within  a  few  days) 

3.  roucine  (next  available  appointment,  within  14  days) 

3.  Please  check  che  appropriate  category  for  your  appointment,  (check  one) 

1. _ scheduled  appointment 

unscheduled;  but  someone  referred  me  (check  one) 

Emprggnrv  Room 

2.  _ another  clinic 

3.  _ other, please  specify: 

unscheduled;  I  tried  to  make  an  appointment  myself,  but  could 
not  get  one 

unscheduled;  did  not  try  to  make  an  appointment;  was  not 
referred 

WALK-IN  PATIENTS  SKI?  TO  QUESTION  ?. 

FOR  PATIENTS  WITH  SCHEDULED  APPOINTMENTS: 

4.  How  long  ago  did  you  schedule  your  appointment? 

1.  today  4.  8-14  days 

2 ♦  1-2  days  5»  15-30  days 

3. _ 3-7  days  6. _ over  30  days 

5.  Would  you  have  liked  an  appointment  sooner  chan  you  were  able  to 
schedule  one? 

1 .  _ yes 

2.  _ no  How  much  sooner? _ 

6.  How  did  you  schedule  chis  appoincmenc?  (check  one) 

1.  _ through  central  appointment  scheduling  (CAS) 

2.  through  the  clinic 

A-l 


7.  By  what  naans  did  you  schedule  it?  (check  one) 


1«  by  CAS  phone-on  post  (in  clinic  areas) 

2.  by  another  phone 

3.  by  nail 

4.  in  person  -  at  clinic  desk 

5.  other,  please  specify 

If  you  nade  the  aopointnent  bv  telephone:  How  many  attempts  to  call 
did  you  nake  before  you  got  an  appointment? 

1.  1  attempt  (got  through  on  first  attempt) 

2.  _ 2  attempts 

3.  3  attempts 

4 .  more  than  3  attempts,  specify  how  many 

5.  can't  remember 


ALL  PATIENTS 


Do  you  have  an  appointment  scheduled  in  another  clinic  for  codav? 


Would  you  have  liked  to  schedule  another  appointment  for  today  if  this 
were  possible? 


10.  Were  your  records  available  for  this  visit  to  the  clinic? 

Medical  Records:  1.  yes 

2.  no 

X-Rays :  1.  yes 


If  either  your  medical  records  or  your  X-rays  or  both  were  available, 
please  tell  us  where  they  were. 


Medical  Records:  1. 

2 ." 


X-Rays : 


_They  were  at  the  clinic  when  I  arrived 
_I  picked  them  up  on  my  way  here 
_I  brought  them  with  me  from  home 

_They  were  at  the  clinic  when  I  arrived 
_I  picked  them  up  on  my  way  here 


12.  Are  you  taking  time  off  from  work  to  come  to  this  appointment? 
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The  following  statements  describe  various  aspects  of  the  outpatien 
clinics  at  KMC.  Please  indicate  how  satisfied  you  are  with  each 
aspect  by  putting  a  check  in  the  column  that  best  describes  your 
opinion. 


Statements 


Ease  of  booking 

appointments . . 

Availability  of  appointments 

at  time  desired . 

The  telephone  system  for 

making  appointments.. . 

Special  instructions  for  pre¬ 
paring  for  appointment . 

Finding  clinic  location . 

Parking . 

Waiting  time  in  clinic . 

Availability  of  medical 

records . 

Attitude  and  helpfulness  of 

appointment  personnel . 

Appointment  scheduling 

system  in  general . 

Overall  experience  with 

care  system . 


Very 

satified 


Somewhat 

satisfied 

2. 


Undecided 


Not  very  N 
satisfied  J  s 
4.  I  5. 


.  Do  you  have  any  additional  comments  or  suggestions 


Please  indicate  your  status,  (check  one) 

1.  _ Active  duty  military 

(military  rank _ 

2.  Retired  military 

3.  Dependent  of  active  duty  military 

4.  _ Dependent  of  retired/deceased 

5.  Civilian  personnel 

6.  Other  (please  specify 


Thank  you  for  taking  the  time  to  fill  out  this  questionnaire. 
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PATIENT  QUESTIONNAIRE 
DENTAL  CLINIC 


The  following  statements  describe  various  aspects  of  the  dental 
clinic  at  KMC*  Please  indicate  how  satisfied  you  are  with  each 
aspect  by  putting  a  check  in  the  column  that  best  describes  your 
opinion. 


Statements 


Ease  of  booking 

appointments . 

Availability  of  appointments 

at  time  desired . 

|  The  telephone  system  for 

making  appointments . 

Special  instructions  for  pre¬ 
paring  for  appointment . 

{Finding  clinic  location...... 

Parking . . 

Waiting  time  in  clinic . . 

■  Availability  of  medical 

records . 

Attitude  and  helpfulness  of 

^  appointment  personnel . 

}  Appointment  scheduling 

system  in  general . 

Overall  experience  with 
care  system . 


Very  Somewhat  Not  very  Not  at  all 

satified  satisfied  Undecided  satisfied  satisfied 

1.  2.  _  3.  U.  5. 


Please  indicate  your  status,  (check  one) 

1.  Active  duty  military 

(military  rank _ 

2.  Retired  military 

3.  _ Dependent  of  active  duty  military 

4.  _ Dependent  of  retired/deceased 

3.  Civilian  personnel 

6.  _ Other  (please  specify _ 


Thank  you  for  taking  the  time  to  fill  out  this  questionnaire. 
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PATIENT  WAITING  TIME  RECORD 


Oitc 


y« 

Clinic  Active  Duty 


Arrival  Tim 
Appointment  Tim 

Tim  vhen  diagnosis/ treatment  begins 


Appointed 
Walk- in 

Thought  had  appolntmnt  but  not  on  schedule 


Its  No 


Medical  record  available  at  first 
contact  with  appointment  desk 

X-Ray  available  at  first  contact 
with  appointment  desk 


HEALTH  CARE  PROVIDER  QUESTIONNAIRE 


1.  Please  write  the  month,  day,  and  year  of  your  birth.  It  will  be  used  as 
an  anonymous  way  to  match  your  responses  to  surveys  done  before  and 
after  Installation  of  the  new  patient  appointment  and  scheduling  system. 


month  day  year 

(If  you  completed  a  questionnaire  regarding  the  manual  appointment 
atid  scheduling  system,  skip  to  question  6.) 

2.  What  is  the  name  of  the  outpatient  clinic  where  you  usually  work? 


3.  Do  you  work  at  other  outpatient  clinics  on  base?  Dyes 

If  yes,  specify  which  one(s)?  _ _ % 


A.  What  functions,  if  any,  do  you  perform  related  to  appointment  scheduling? 
Check  all  that  apply. 

_ 1)  refer  patients  for  appointments 

_ 2)  participate  in  establishing  monthly  appointment  schedules 

_ 3)  make  appointments  for  patients  I  am  referring  to  other  clinics 

_ _ 4)  other  (specify)  _ _ _ 

_ 5)  do  not  perform  any  functions  related  to  patient  appointment 

scheduling 
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5.  In  your  view,  how  important  is  the  patient  appointment  and  scheduling 
•ystem  to  the  quality  of  care  at  KMC?  Check  one. 

_ 1)  very  important 

_ 2)  important 

3)  undecided,  no  opinion 

A)  not  very  important 

•  5)  not  at  all  important 

6.  We  would  now  like  your  opinion  about  the  present  appointment  and  scheduling 
system.  For  each  of  the  following  statements,  please  tell  us  how  welL  the 
current  system  generally  performs. 


a)  Percent  of  care  providers  who 
express  satisfaction  with  the 
overall  quality  of  appointment 

and  scheduling  service.  1) 

b)  Percent  of  scheduled  patients 

who  keep  their  appointments  2)_ 

c)  Percent  of  scheduled  patients 
who  have  X-rays  available 

for  their  appointment.  3)_ 

d)  Percent  of  scheduled  patients 

•  who  are  appropriately  scheduled 
at  the  outpatient  clinic  where 
you  usually  work.  (This  state¬ 
ment  refers  exclusively  to 
clinical  "appropriateness").  A) 

e)  Percent  of  scheduled  patients 
who  have  a  medical  record 
available  for  their  appointment.  5) 


The  following  statements  describe  various  aspects  of  the  scheduling  system. 
Please  Indicate  how  satisfied  you  are  with  each  aspect  of  the  current  system 
by  placing  an  "X"  under  the  column  that  best  describes  your  opinion. 


Very  Somewhat  Undecided/  Not  Very  Not  at  all 

Satisfied  Satisfied  No  Opinion  Satisfied  Satisfied 

a)  Patient 's  ease  of 

booking  appointments.  •'  1. _  2. _  3. _  4. _  5. _ _ 


b)  Patient's  ability  to  book 
appointments  at  time 

desired.  1. _  2. _  3. _  4. 

c)  Waiting  time  after 

'patient  arrives.  1. _  2. _  3. _  4. 


d)  Availability  of  medical 
records  at  time  of 
appointment  at  your 

'clinic.  1. _  2.  _  3.  4. 


e)  Telephone  system  for 

making  appointments.  1. _  2. _  3. _  4. 


f)  Availability  of 
x-rays  at  time  of 
appointment  at  your 

clinic.  1. _  2. _  3.  4. 


g)  Accuracy  of  assigning 
patients  to  appropriate 

clinic.  1. _  2. _  3. _ _  4. 


h)  Ability  of  appointment  and  scheduling  to  accommodate : 

Flexible  schedules  l. _  2. _  3. _  4. 


Cancellations  1. _  2. _ 3. 


Changes  in  provider 
schedules 


2. _  3. _  4. 


Same  day  multiple 
clinic  visits 


1. 


2 


3. 


4. 


Special  instructions  that 
oust  be  given  to  patients 
prior  to  their  vlsists.  1 


Patients  requiring  special 
processing  and  care.  1 


Patients  requiring 
immediate  appoint¬ 
ments.  1 

i)  Overall  quality 
of  appointment  and 
scheduling  service.  1 


APPOINTMENT  AND  SCHEDULING  SUPPORT  STAFF  QUESTIONNAIRE 


Please  write  the  month,  day,  and  year  of  your  birth.  It  will  be  used 
in  an  anonymous  way  to  match  your  responses  to  surveys  performed  before 
and  after  installation  of  the  new  patient  appointment  and  scheduling 
system. 


Month/Day/Year 


(IF  YOU  COMPLETED  A  QUESTIONNAIRE  REGARDING  THE  MANUAL  SYSTEM, 
SKIP  TO  QUESTION  6.) 

2.  Indicate  whether  you  work  in  CAS  or  an  outpatient  clinic.  If  you 
work  in  an  outpatient  clinic  indicate  the  name  of  the  clinic. 


1. _ 

CAS 

2. _ 

Outpatient  Clinic: 

Name  of  Clinic 


3.  What  funcitons,  if  any,  do  you  perform  related  to  appointment 
scheduling?  Check  all  that  apply. 

_ _l)  refer  patients  for  appointments 

2)  participate  in  establishing  monthly  appointment  schedule 

3)  make  patient  appointments  (by  telephone  or  at  the  clinic) 
_ __4)  verify  (confirm)  appointments 

_ 5)  cancel  appointments 

6)  do  not  perform  any  functions  related  to  patient  appointment 
scheduling 

_ 7)  other  (specify) _ * _ 


4.  In  your  view,  how  important  is  the  patient  appointment  and  scheduling  syst 
to  the  quality  of  care  at  KMC?  Check  one. 


1)  very  important 

2)  somewhat  important 

3)  undecided,  no  opinion 
A)  not  very  important 
.5)  not  at  all  important 
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We  would  also  like  your  opinions  about  how  the  present  appointment  and 
scheduling  system  performs.  For  each  of  the  following;  measures  of  per¬ 
formance,  please  indicate  how  well  you  think  tlu*  current  Hystem  lias  done. 


EVALUATION  OF  THE  CURRENT 

PERFORMANCE  MEASURE  APPOINTMENTS  A  SCHEDULING  SYSTEM 

a)  Percent  of  scheduling  and 
support  staff  expressing 
satisfaction  with  the  overall 
quality  of  appointment  and 

scheduling  service .  _ X 

b)  Percent  of  the  scheduled  patients 

who  keep  their  appointments  _ X 

6.  Please  indicate  how  satisfied  you  are  with  each  of  the  following  aspects 
of  the  current  appointment  and  scheduling  system  by  placing  an  "X"  under 
the  column  that  best  describes  your  opinion. 


Very 

Satisf led 

Somewhat 
Satisf ied 

Undecided/ 

No  opinion 

Not  Very 
Satisfied 

Not  at  all 
Satisf ied 

Patients  ease  of 
booking  appointments 

1. 

2. 

3. 

A. 

5. 

Patient's  ability  to 
book  appointments  at 
time  desired. 

1. 

2. 

3. 

4. 

3. 

Telephone  system  for 
making  appointments 

1. 

2. 

3. 

4. 

5. 

Ability  of  appointment  and  scheduling  system 

to  accommodate^ 

• 

1.  Cancellations 

1. 

2. 

3. 

4. 

5. 

2.  Changes  in  pro¬ 
vider  schedules 

1. 

2. 

3. 

4. 

5. 

3.  Same  day  multiple 
clinic  visits 

1. 

2. 

3. 

4. 

5. 

A.  Special  instructions 
that  must  be  given 
to  patients  prior  to 
their  visits.  1  • 

2. 

3. 

4. 

5. 
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Very  Somewhat  Undecided/  Not  Very  Not  at  ai 

Satisfied  Satisfied  No  Opinion  Safisfied  Satisfied 


5.  Patients  requiring 

immediate  appoint¬ 
ments  1. _  2. 

6.  Patients  requiring 
special  processing 

end  qare.  1. _  2. 

7.  Appointments  which 
•  vary  in  length 

(e.g. ,  15  vs.  30 


vs.  45  minutes).  1. _  2 ._ 

• 

8.  Ability  of  patients 
to  obtain  appointments 
by  mall.  1. _  2 . 

e)  Overall  quality  of 

appointment  and 

scheduling  service.  1. _  2. 

f)  Relative  ease  for  you  to  accommodate: 

1.  Appointing  patients  1. _  2. 

2.  Cancelling  appoint¬ 
ments  l. _  2. 


3. 


3. 


3. 


3. 


3. 


3. 


3. 


4. _  5. 


4. _  5. 


4.  5. 


4.  5. 


4. _  5. 


4. _  5. 


4.  _  5. 


3.  Rescheduling 

appointments  1. _  2. _  3. _  4. _  5. 

4.  Dispensing 
information  to 

patients  1. _  2. _  3. _  4. _  5. 


'■i 
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PROVIDER  NAME _  CLINIC 

PHONE  CONSULT  RECORD 

(Please  check  (  )  each  time  a  phone  consult  is 


a*  -n 


Clinic : 


PROVIDER  INTERVIEW  FORM 


1.  During  a  typical  work  week,  how  many  hours  are  you  assigned  to 

this  clinic?  _ _ 

2.  During  this  time,  how  many  patients  do  you  see  per  day  and  per  week? 

Day  Week 

3.  Of  the  patients  seen  by  you,  what  percentage  are  appointed  patients 
and  what  percentage  are  walk-ins? 


Appointed' 


Walk-ins 


4.  How  much  "downtime"  do  you  have  in  a  typical  work  day? 

i  i 

How  do  you  use  this  time? _ 


5.  Are  you  satisfied  with  the  way  the  appointment  and  scheduling  staff 
"book"  your  day?  _ Yes  _ NO 

If  no,  explain: _ 


Do  appointment  and  scheduling  staff  ever  double-book  your  appointment 
time?  _ Yes  _ No 

If  yes,  how  do  you  handle  this? _ _ _ 


7.  How  long  do  patients  usually  wait  in  the  clinic  to  see  you? 

_  Is  this  an  acceptable  amount  of  time,  in  your  opinion? 


1 
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8.  How  do  you  handle  active  duty  appointments  and  other  triaging 
situations? 


9.  Do  you  feel  that  the  installation  of  the  automated  appointment  and 
•scheduling  system  has  improved  your  efficiency,  productivity, 
and  tne  quality  of  care  you  provide?  If  so,  what  benefits  have 
you  realized  as  a  result  of  this  system? 


PATIENT  TRACKING  FORM 


Specialty  Clinic 


Orthopedics 


General  Surgery 


1*  Date  of  call  to  Central  Appointments  for  an  appointment  in 
Primary  Care: _ 

2.  Date  of  appointment  in  Primary  Care: _ 

3.  Date  of  contact  with  Orthopedics  or  General  Surgery  to  request 

an  appointment: _ 

4.  Date  of  appointment  in  either  Orthopedics  or  General  Surgery: 


5.  Between  the  date  you  visited  Primary  Care  and  the  date  you  visited 

Orthopedics  or  General  Surgery,  were  you  referred  to  any  other 
clinic  or  division  of  the  hospital?  Yes _  No _ 

6.  (If  "yes"  to  Question  5)  What  division (s)  were  you  referred  to 
and  what  were  the  dates  of  contact? 


% 

Division 

Date 

Division 

Date 

Division 

Date 

R 

7.  (If  "yes"  in  Questions) 
appropriate?  Yes 

Do  you  feel  that  this 
No 

If  no,  why  not? 

Were  you  satisfied  with  the  time  it  took  you  to  get  an  appointment 
with  either  Orthopedics  or  General  Surgery?  Yes _  No _ 

If  no,  explain: _ -m _ 


9.  Do  you  feel  it  would  have  been  better  to  bypass  Primary  Care  screening? 
Yes _  No _ 


If  yes,  why? 


